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Summary 

Systematic  methods  for  training  system  design  have  been  available  for  years.  Using  such  methods  for  the 
development  of  user  requirements  and  specifications  for  technically  advanced  training  means  (simulators,  E- 
leaming,  CBT,  etc.)  ensures  that  these  will  fulfil  the  training  needs.  The  problem  is  that  applying  systematic 
methods  takes  time  and  effort,  and  requires  expertise.  Our  solution  is  to  work  iteratively  and  in  workshops 
under  supervision  of  experienced  facilitators.  The  same  systematic  method  for  training  system  design  can  be 
used  at  different  moments  with  different  levels  of  detail.  The  SLIM  method  has  been  designed  for  the  needs 
statement  phase.  In  this  early  stage  training  needs  analysis  and  training  program  design  are  done  at  a  rather 
global  level.  Later  on,  using  the  same  or  a  similar  step-by-step  method,  the  results  can  be  elaborated  in  more 
detail,  in  order  to  write  specifications  and  -eventually-  to  implement  training.  The  SLIM  method  emphasises 
aspects  that  are  often  not  systematically  addressed:  the  place  of  training  means  in  overall  training,  the  choice 
between  high  and  lower  fidelity  options,  facilities  for  instruction,  feedback  and  assessment,  and  the  role  of 
the  instructors.  Inviting  all  stakeholders  ensures  that  the  chosen  solution  is  widely  supported.  In  this  paper 
we  briefly  describe  the  SLIM  method,  and  the  experiences  during  its  application  in  a  number  of  different 
cases.  We  conclude  that  needs  statements  can  be  developed  quickly  and  systematically  using  the  SLIM 
method  in  workshop  setting. 


i 

Parts  of  this  paper  will  also  appear  in  Verstegen  (in  press). 
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1.  Background 

In  the  Royal  Netherlands  Army  (RNLA)  the  development  of  a  needs  statement  is  the  first  phase  in  the 
procurement  of  technically  advanced  training  means  such  as  simulators,  Computer  Based  Training  (CBT) 
web-based  learning  environments  and  Virtual  Environments.  A  needs  statement  describes  a  preliminary 
selection  of  training  means  and  a  first  estimate  of  how  much  they  will  cost.  Of  course,  it  will  have  to  be 
elaborated,  adapted  and  revised  later  on.  A  well  motivated  needs  statement  is  important,  however,  as 
financial,  and  other  resources  for  these  expensive  training  means  have  to  be  reserved  well  in  advance 
(Verstegen,  Barnard,  and  van  Rooij,  1999).  Based  on  discussions  with  stakeholders  from  the  RNLA,  the 
assignment  for  TNO  Human  Factors  (TNO-HF)  was  to  develop  a  method  to  be  used  to  develop  needs 
statements,  or  more  specifically,  to  specify  a  way  to  determine  the  present  and  future  training  needs  within  a 
limited  time  span  and  based  on  information  that  can  be  incomplete  or  insecure,  to  decide  whether  and  why 
the  acquisition  of  advanced  training  means  will  be  necessary,  to  estimate  roughly  the  resources  that  should 
be  allocated  for  this  purpose  and  which  other  consequences  (organisation,  logistic  and  personnel)  have  to  be 
taken  into  account. 

Field  research  suggests  that  currently  training  needs  are  often  not  the  major  focus  during  the  design  of 
training  systems,  possibly  because  those  responsible  for  their  specification  are  usually  operational  staff  or  the 
designers  of  the  operational  systems  and  not  the  people  who  are  responsible  for  training.  When  instructors 
are  involved  they  are  Subject  Matter  Experts  (SME)  who  have  little  didactic  background  and  experience  with 
instructional  design.  The  result  is  that  specifications  are  often  technology-pushed  and  product-oriented,  i.e. 
written  in  terms  of  a  simulation  of  the  operational  system  or  describing  an  existing  training  system  seen 
elsewhere.  In  other  cases,  the  functional  specifications  remain  vague  for  a  long  time,  and  elaboration  in  more 
detail  is  postponed.  This  may  cause  unacceptable  delays  in  the  availability  of  training  facilities  (Farmer, 
Joma,  Riemersma,  van  Rooij  and  Moraal,  2000;  Verstegen,  Barnard  and  van  Rooij,  1999;  Wallace  and 
Northham,  1998).  Using  a  systematic  method  can  help  to  avoid  these  problems. 

Different  kinds  of  systematic  instructional  design  and  development  methods  are  described  in  the  literature 
(see  e.g.  Verstegen,  in  press,  for  an  overview).  None  of  these  methods  was  directly  applicable,  however, 
because  they  are  not  specifically  geared  towards  this  early  phase  in  the  acquisition  process,  not  adapted  to 
the  context  of  the  RNLA  organisation  and  not  integrated  with  other  RNLA  procedures.  After  an  inventory  of 
available  method  and  tools,  we  evaluated  two  of  them  with  RNLA  target  users:  the  MASTER  method 
(Farmer  et  al.  1999)2  and  the  BOOT  decision-making  support  tool  (van  der  Hulst,  de  Hoog  &  Wielemaker, 
1999)3.  The  results  are  described  in  van  der  Hulst  and  Verstegen  (2000).  The  conclusions  were  that  the 
MASTER  method  with  its  many  steps  and  sub-steps  is  too  detailed  for  the  development  of  needs  statements. 
However,  the  overall  structure  of  the  method  is  a  suitable  framework.  The  second  half  of  the  method  is 
specifically  geared  towards  the  specification  of  training  simulators,  but  it  seems  that  the  specification  of 
other  kinds  of  training  means  can  take  place  along  the  same  lines,  especially  in  this  early  phase.  Like  the 
MASTER  method,  already  existing  RNLA  procedures,  for  example  for  the  development  and  implementation 
of  courses,  and  the  design  of  other  kinds  of  training  means,  are  based  on  an  Instructional  System  Design 
(ISD)  approach  (see  e.g.  Gagne,  Briggs  and  Wager,  1992).  Therefore,  we  did  not  foresee  major  integration 
problems.  The  kind  of  explicit  decision-making  support  offered  by  BOOT  was  considered  a  valuable 
addition,  and  was  initially  implemented  by  defining  explicit  decision  points  with  accompanying  paper-based 
checklists.  In  future,  software  tools  to  support  decision-making  and/or  expert  systems  to  for  specific  decision 
points  may  be  developed. 

One  of  the  problems  of  the  ISD-based  approach  is  that  it  takes  a  considerable  amount  of  time  and  effort  to 
get  insight  in  training  system  specifications.  If  it  is  used  too  early  in  the  acquisition  trajectory  the  invested 


2  The  MASTER  method  for  the  specification  of  training  simulators  was  developed  within  the  European  defense 
research  project  MASTER  (Military  Applications  of  Simulator  and  Training  concepts  based  on  Empirical  Research, 
EUCLID,  RTP  11.1). 

3  BOOT  stands  for  decision  support  for  the  selection  of  training  means  (in  Dutch). 
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time  and  effort  may  be  wasted,  e.g.  because  the  project  does  not  get  funding  or  because  conditions  change. 
Waiting  too  long,  however,  means  that  there  is  not  enough  time  left  for  thorough  analysis  of  training  needs 
and  alternative  training  solutions.  Our  solution  is  to  work  iteratively,  at  different  levels  of  detail.  In  the  early 
stages,  training  needs  analysis  training  program  design  and  training  system  specification  are  done  at  a  rather 
global  level,  sufficient  to  write  a  well  motivated  needs  statement.  Later  on,  going  through  the  same  or  a 
similar  sequence  of  steps,  the  results  can  be  further  elaborated. 

Another  problem  with  ISD-based  methods  is  that  they  are  described  in  a  rigid,  linear  way  and  need  to  be 
adapted  to  the  design  project  context.  Descriptive  research  indicates  that  this  is  not  necessarily  a  problem  for 
professional  instructional  designers,  but  is  difficult  for  novices.  Moreover,  the  development  of  needs 
statements  for  advanced  training  means  is  a  long,  complex  process  involving  several  parties  with  different 
backgrounds  and  (sometimes  conflicting)  interests  and  different  kinds  of  (often  contradicting)  constraints. 
Our  solution  for  these  problems  is  to  invite  all  important  stakeholders  to  workshops,  which  take  place  under 
supervision  of  an  experienced  instructional  designer  who  organises  and  monitors  the  design  process, 
applying  the  systematic  method  described  Section  2.  The  participants  bring  in  their  own  information  and 
expertise  and  they  eventually  take  the  decisions  collectively,  thus  ensuring  that  the  designed  solution  will  be 
acceptable  for  all  stakeholders. 

In  Section  2  we  will  describe  our  method  for  the  development  of  needs  statements.  The  method  has  been 
applied  in  five  different  cases.  We  will  reflect  on  our  experiences  during  these  projects  in  Section  3,  and 
present  some  conclusions  in  Section  4. 


2.  The  SLIM  method 

The  SLIM  (Specifying  Learning  means  in  an  Iterative  Manner)  method  is  based  on  the  MASTER  method, 
but  adapted  for  the  needs  statement  phase  and  the  first  phase  in  the  acquisition  process.  The  terminology  and 
names  of  steps  have  been  customised  for  the  RNLA.  The  SLIM  method  emphasises  aspects  that  are  currently 
often  not  systematically  addressed:  the  place  of  training  means  in  overall  training,  the  choice  between  high 
and  lower  fidelity  options,  facilities  for  instruction,  feedback  and  assessment,  and  the  role  of  the  instructors. 

2.1  The  steps  of  the  SLIM  method 

The  SLIM  method  leads  users  through  a  global  analysis  and  design  process  step-by-step.  This  does  not, 
however,  mean  that  the  process  will  always  be  the  same.  Depending  on,  for  example,  the  complexity  of  the 
domain  and  the  amount  of  information  available,  steps  may  take  more  or  less  time,  some  steps  may 
sometimes  get  less  emphasis  or  may  only  be  partly  executed.  The  method  consists  of  four  main  phases,  all 
divided  in  sub-steps  (see  Figure  1): 

I.  Problem  definition:  inventory  of  the  intended  outcome/results,  the  demands,  available  resources  and 
project  risks. 

II.  Analysis  of  training  needs:  global  analysis  of  needs  for  education  and  training. 

III.  Design  blue  print  &  Selection  of  training  means:  design  first  draft  of  training  and  select  a  suitable 
combination  of  training  means. 

IV.  Design  of  specifications  &  Cost  estimation:  definition  of  global  specifications  and  estimation  of  the 
resources  that  will  be  required  for  the  acquisition  and  use  of  the  training  means. 

Developing  a  complete  and  thorough  needs  statement  takes  time  and  manpower.  Therefore,  the  SLIM 
method  starts  with  an  inventory  of  the  situation  in  order  to  decide  whether  it  is  worthwhile  to  invest  in  such 
an  undertaking.  This  phase  is  meant  as  an  explicit  analysis  of  the  problem  situation,  as  has  been  observed  in 
the  behaviour  of  expert  instructional  designers  (e.g.  Rowland,  1992,  1991;  Goel  and  Pirolli,  1992,  1989). 
Expert  designers  have  been  observed  to  generate  several  potential  solutions,  and  not  commit  to  one  particular 
solution  until  later  on.  This  kind  of  behaviour  is  stimulated  by  asking  users  to  sketch  a  number  of  likely 
training  solutions  in  general  terms  up  front,  during  the  first  phase  (see  also  the  discussion  below).  In  the 
second  phase  a  global  analysis  of  the  needs  for  education  and  training  is  executed,  and  in  the  third  phase  a 
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first  draft  of  the  course  or  training  program  is  designed.  For  the  part  that  will  be  executed  with  advanced 
training  means,  prototypical  lessons/scenarios  and  types  of  instruction  and  feedback  are  specified.  This 
information  is  used  in  the  fourth  phase  to  determine  global  specifications  for  the  required  training  means 
(primary  user  requirements)  and  other  consequences  for  the  training  environment,  such  as  the  required 
number  and  capabilities  of  instructors,  and  the  additional  hardware,  software  and  technical  support  required 
for  training.  This  results  in  a  rough  estimate  of  the  required  resources.  Three  kinds  of  resources  play  a  role 
here:  resources  for  the  acquisition  of  training  means,  resources  for  the  use  and  maintenance  of  those  products 
in  future  training,  and  resources  for  the  development  of  (detailed)  specifications.  Figure  1  provides  an 
overview  of  the  four  phases  of  the  SLIM  method  and  their  steps. 

The  intention  of  the  SLIM  method  is  not  to  develop  final  and  fine  grained  specifications,  but  to  start  thinking 
about  how  education  and  training  will  probably  take  place  and  which  kind  of  training  means  will  probably  be 
used.  On  this  basis  a  reasonably  realistic  cost  estimation  can  be  made,  taking  several  alternatives  into 
account.  In  principle,  the  same  steps  of  the  SLIM  method  can  be  executed  at  a  more  detailed  level  later  on  in 
the  acquisition  trajectory.  We  will  come  back  to  this  issue  in  Section  2.4. 

2.2  Specific  aspects  of  the  SLIM  method 

The  SLIM  method  proposes  solutions  for  some  of  the  problems  with  the  MASTER  method  encountered 
during  previous  evaluation  studies  (Verstegen,  in  press).  We  will  discuss  some  of  these  issues  below: 

a)  Dealing  with  resources 

The  MASTER  method  is  not  entirely  clear  about  when  and  how  designers  should  deal  with  information 
about  the  resources  available  for  the  acquisition  and  use  of  training  means.  The  SLIM  method  starts  with  an 
explicit  inventory  of  demands,  resources  and  risks  (Step  1-2).  These  are  stored  in  three  lists  are  kept  available 
at  all  times.  After  each  step  users  are  reminded  explicitly  to  keep  these  lists  up  to  date. 

b)  Generating  alternatives 

As  discussed  above,  users  are  asked  to  sketch  a  number  of  possible  solutions  in  general  terms  up  front.  This 
is  part  of  Step  1-1  and  is  meant  both  to  make  the  participants'  preconceived  ideas  explicit  and  as  an  inventory 
of  alternative  solutions.  Comparing  alternative  solutions  comes  back  as  an  explicit  step  half  way  through  the 
design  phase  (see  below,  under  point  d).  For  now,  it  is  up  to  the  workshop  facilitators  to  make  sure  that  the 
participants  keep  on  considering  feasible  alternatives  during  the  rest  of  the  design  process  (see  Section  4). 

c)  Explicit  decision  points 

After  each  step  an  explicit  decision  point  is  introduced.  A  general  checklist  with  recurring  questions  or 
considerations  has  been  developed  (e.g.  "Are  your  assumptions  still  correct?"),  and  at  important  decision 
points  more  specific  questions  are  asked  (e.g.  after  the  selection  of  training  means  users  are  prompted  to 
consider  possible  alternative  solutions). 

d)  Prioritising  and  weighing  alternatives 

To  encourage  users  to  consider  alternatives  they  are  explicitly  asked  to  select  a  wide  range  of  suitable 
training  means  for  each  learning  activity,  and  to  prioritise  these  options  from  most  desirable  to  least  desirable 
(this  is  part  of  Step  III-2).  Subsequently,  users  select  a  combination  of  training  means  with  which  all  learning 
activities  can  be  executed,  though  maybe  not  all  in  the  most  preferred  way.  Users  are  advised  to  envision 
alternative  combinations  and  to  define  their  advantages  and  disadvantages,  before  making  a  choice.  Later  on 
in  the  acquisition  trajectory  these  alternatives  can  be  reconsidered,  for  example  when  the  resources  do  not 
permit  the  acquisition  of  (a  part  of)  the  initially  chosen  set  of  training  means. 
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Figure  1: 


The  steps  of  the  SLIM  method  for  the  development  of  needs  statements  for  technically  advanced  training  means. 


RTO-MP-HFM-101:  Paper  24 


Page  5  of  1 1 


ORGANIZATION 


e)  Assessment  of  the  trainees'  performance 

Evaluations  with  the  MASTER  method  indicated  that  assessment  planning  without  an  overview  of  the  whole 
course  is  difficult,  and  that  there  is  some  overlap  between  assessment  and  debriefings  (Verstegen,  in  press). 

In  the  SLIM  method  another  point  of  view  has  been  chosen:  assessments  are  regarded  as  a  special  category 
of  learning  activity  and  are  planned  in  the  same  step.  Subsequently,  they  are  elaborated  in  the  same  way,  i.e. 
a  range  of  suitable  training  means  is  chosen  and,  if  the  assessment  activity  is  allocated  to  advanced  training 
means,  a  'scenario'  for  the  assessment  is  described  in  step  111-3. 

2.3  Applying  the  SLIM  method  in  workshop  setting 

The  SLIM  method  has  been  applied  in  the  form  of  workshops  with  all  stakeholders  under  supervision  of  two 
experienced  facilitators.  In  this  setting,  the  workshop  leader  is  fully  responsible  for  the  management  of  the 
design  process:  explaining  the  goals  of  the  SLIM  method,  organising  the  process,  structuring  and  guiding  the 
discussions,  making  sure  that  no  available  information  is  neglected  and  that  all  feasible  alternatives  were 
taken  into  account,  deciding  when  to  go  to  the  next  step  and  also  when  iteration  is  necessary.  A  second 
facilitator  takes  notes  and  is  responsible  for  the  documentation  of  the  entire  process. 

The  role  of  the  workshop  participants  is  to  provide  the  necessary  problem  and  domain  information,  and  to 
make  decisions.  Ideally,  the  number  of  participants  is  between  four  and  eight,  including  at  least  a  domain 
expert,  a  course  designer,  a  future  instructor  and  a  representative  from  the  school's  management.  It  is  also 
possible  to  invite  other  stakeholders,  such  as  a  representative  of  the  operational  unit(s)  where  trainees  will 
work  in  future,  a  CBT  developer,  or  financial  and  technical  experts.  Too  many  participants  is  expected  to 
make  the  workshops  inefficient.  On  the  other  hand,  inviting  all  stakeholders  ensures  a  widely  accepted 
solution. 

Iteration  is  encouraged  by  organising  a  sequence  of  workshops.  In  between  workshops  the  participants  can 
reflect  on  the  results  so  far,  collect  additional  information  and  discuss  with  their  colleagues.  New  information 
or  discussion  points  are  input  for  the  next  workshop,  and  thus  taken  into  account  immediately. 

2.4  Reusing  the  results  of  the  SLIM  method 

The  SLIM  method  is  meant  to  be  used  at  a  global  level  of  detail.  Initially  training  needs  analysis,  training 
program  design,  and  training  system  specification  are  executed  at  a  rather  global  level,  sufficient  to  write  a 
well  motivated  needs  statement.  The  description  of  steps,  the  guidelines,  checklists  and  other  support  are 
focussed  on  helping  users  to  go  through  the  steps  in  a  limited  amount  of  time  (allowing  iterations,  see 
Section  2.3).  At  the  same  time  it  ensures  that  the  information  collected  during  the  first  phase  of  the 
acquisition  process  can  be  reused  later  on  when  specifications  are  developed  at  a  more  detailed  level  as 
depicted  in  Figure  2. 

In  principle,  the  same  steps  of  the  SLIM  method  can  be  executed  at  a  more  detailed  level  later  on  in  the 
acquisition  trajectory,  e.g.  to  develop  functional  specifications,  technical  specifications  and,  eventually,  to 
develop  and  implement  training.  Alternatively  users  can  choose  for  a  more  specific  and  detailed  ISD-based 
method,  such  as  the  MASTER  method  for  developing  simulator  specifications. 
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Figure  2:  Reuse  of  information  collected  with  the  SLIM  method. 


3.  Experiences  with  the  SLIM  method 

In  this  section  we  will  report  our  experiences  with  the  SLIM  method.  After  discussing  the  five  cases  in 
Section  3. 1  we  will  discuss  how  the  SLIM  method  was  applied  and  adapted  in  Section  3.2,  reflect  on  the  role 
of  the  facilitators  in  Section  3.3,  and  briefly  address  iteration  and  dealing  with  uncertainty  in  Section  3.4. 


3.1.  The  five  cases 


We  have  applied  the  SLIM  method  in  five  different  cases  with  customers,  three  times  for  the  RNLA,  once 
for  the  Royal  Netherlands  Navy  and  once  for  the  Royal  Netherlands  Air  Force.  For  the  last  two  cases,  we 
adapted  the  terminology  and  the  names  of  the  steps  to  those  used  in  these  organisations.  The  results 
presented  below  are  based  on  the  written  reports  that  were  made  of  all  the  sessions  (Boot,  Verstegen  and 
Veerman,  2002;  Melis  and  van  Berio,  2003;  Verstegen,  Veerman  and  van  der  Arend,  2001),  the  discussions 
with  the  participants  during  the  sessions,  and  the  experiences  of  the  facilitators. 

In  three  of  the  cases,  the  resulting  needs  statement  describes  a  CBT  application,  in  one  case  a  desktop-based 
simulation  and  in  one  case  a  simulator  (all  to  be  used  in  combination  with  other  training  means).  All  five 
cases  were  complex,  but  for  different  reasons.  In  one  case  the  main  source  of  complexity  was  the  fact  that  the 
operational  system  did  not  yet  exist,  and  that  the  information  about  the  exact  nature  of  the  tasks  that  trainees 
would  have  to  be  trained  for  was  incomplete  and  insecure.  In  another  case  the  tasks  were  relatively  easy,  but 
the  complexity  from  an  educational  point  of  view  was  caused  by  the  very  large  number  of  trainees  to  be 
trained  for  different  subsets  of  tasks.  In  two  cases,  the  same  training  means  would  have  to  be  used  by  clients 
from  two  rather  different  organisations  who  were  planning  to  use  the  same  operational  system  in  different 
ways;  there  were  also  some  differences  in  organisational  culture,  capabilities  of  instructors  and  the 
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characteristics  of  target  trainees.  In  both  these  cases,  an  additional  complication  was  that  the  assignment  of 
the  participants  of  the  workshop  was  to  design  CBT  to  be  used  in  combination  with  a  simulator  that  was 
being  specified  at  the  same  moment  by  a  different  group  (with  some  overlap  in  participants).  And  in  one 
case,  the  course  addressed  during  the  workshop  was  meant  to  be  just  an  example  to  investigate  the  potential 
role  of  simulation-based  CBT.  The  challenge  in  this  case  was  to  incorporate  an  existing  instructional  design 
model  for  maintenance  training  and  to  replace  (part  of)  the  practice  with  simulators  and/or  operational 
systems  with  CBT-based  exercises. 

Since  the  complexity  of  the  cases  differed,  the  amount  of  time  required  to  execute  the  SLIM  method  differed 
as  well.  The  case  studies  took  four  to  seven  days  divided  over  two  to  four  sessions.  In  between  sessions  the 
participants  went  back  to  their  workplace  to  collect  more  information  and  discuss  the  results  with  their 
colleagues.  In  some  cases,  the  workshop  participants  took  away  'homework',  i.e.  further  elaboration  of  steps 
to  be  done  independently  in  between  sessions.  In  other  cases  participants  were  not  able  and/or  willing  to  do 
this.  It  should  be  noted  that  the  level  of  detail  of  the  results  differed  as  well,  mainly  because  the  amount  of 
time  available  for  workshops  was  limited  by  project  boundaries. 

3.2.  The  SLIM  method  applied  to  different  cases 

We  had  expected  to  have  to  adapt  terminology  for  non-RNLA  participants  (see  above).  However,  during  the 
workshops  it  became  clear  that  in  all  cases  participants  misinterpreted  some  terms,  or  used  them  in  slightly 
different  ways.  When  miscommunications  hindered  the  process,  terms  were  redefined  on  the  fly  or  replaced 
by  terms  that  were  better  known  by  the  participants.  In  some  cases  the  method  was  slightly  customised  as 
well,  e.g.  combining  steps  three  and  four  in  the  design  phase  in  a  less  complex  case,  or  leaving  out  the  fourth 
phase  because  it  was  not  the  responsibility  of  the  workshop's  participants.  This  ad-hoc  customisation  was 
experienced  as  an  advantage  of  applying  the  SLIM  method  in  a  workshop  setting  in  direct  contact  with  the 
stakeholders,  but  it  requires  considerable  effort  and  skill  from  the  facilitators  (see  Section  3.3). 

Applied  in  this  way,  the  SLIM  method  proved  to  be  robust  and  versatile  enough  for  all  five  cases.  In  the 
experience  of  the  facilitators  the  method  was  a  valuable  instrument  in  structuring  the  development  process 
and  maintaining  focus  and  consistency.  The  participants  valued  the  structured  approach  of  the  SLIM  method, 
but  clearly  preferred  not  to  be  bothered  with  details  regarding  the  SLIM  method  or  instructional  design  and 
development  theory  in  general. 

Note  that  in  this  format,  the  participants  take  all  the  final  decisions.  This  means  that  the  SLIM  method  works 
towards  a  solution  that  suits  the  organisation  and  is  accepted  by  all  stakeholders,  which  can  be  -but  is  not 
necessarily-  innovative.  The  acquisition  of  training  means  had  to  solve  existing  or  expected  problems  of  the 
organisation.  Sometimes  new  types  of  training  means  were  required,  e.g.  because  the  participants  expected 
that  they  would  have  much  less  opportunity  to  let  trainees  practise  with  the  operational  system  in  future.  In 
one  case,  innovation  was  explicitly  the  goal  of  the  workshops  (i.e.  simulation-based  CBT,  see  above).  In 
another  case,  however,  the  participants  decided  that  for  a  new  course  classroom  based  lessons  were  most 
suitable,  much  like  the  classroom  based  lessons  in  existing  courses  at  the  school.  Another  observation  was 
that  the  participation  of  all  stakeholders  is  vital.  In  one  of  the  cases  not  all  stakeholders  were  present  during 
the  workshops,  which  lead  to  uncertainty  and  speculation  about  'what  they  would  want',  and  extra  time 
investment  in  reporting  the  results  of  the  workshops,  getting  feedback  and  adapting  the  results  accordingly. 

3.3.  The  role  of  the  facilitators 

The  role  of  the  facilitators  is  important.  The  workshop  leader  needs  to  be  familiar  with  instructional  design 
and  development  models  in  general  and  with  the  details  of  the  SLIM  method,  since  he  or  she  has  to  take  care 
of  organising  and  managing  the  process.  The  interaction  can  be  quite  unorganised,  sometimes  almost  chaotic, 
since  the  workshop  participants  do  not  (and  are  not  meant  to)  pay  attention  to  the  SLIM  method  and  the 
development  process.  It  is  the  task  of  the  workshop  leader  to  introduce  the  different  activities  (i.e.  steps  of 
the  SLIM  method)  and  to  help  the  participants  to  complete  the  different  tasks.  The  workshop  leader  also  has 
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to  maintain  an  overview  of  what  needs  to  be  done,  and  decide  when  to  go  to  the  next  step  or  back  to  a 
previous  step.  On  top  of  that,  the  workshop  leader  decides  if  the  terminology  and  the  SLIM  method  need  be 
customised,  and  how  this  can  be  done  without  violating  the  important  principles  underlying  the  method.  The 
second  facilitator,  who  is  responsible  for  the  report,  also  needs  to  be  familiar  with  the  SLIM  method  in  order 
to  be  able  to  summarise  and  reorganise  data  that  may  result  from  different  moments  in  time  according  to  the 
steps  of  the  method. 

In  most  cases  there  were  participants  with  different  backgrounds  and  conflicting  interests,  and  some 
oveipowering  personalities.  Participants  have  different  positions  in  the  organisation's  hierarchy,  and  often 
existing  political  problems  played  a  role.  The  workshop  format  proved  to  be  an  efficient  way  to  collect  all 
available  information,  make  the  different  opinions  and  arguments  of  stakeholders  explicit,  and  come  to  joint 
decisions.  The  workshop  leader,  however,  needs  additional  skills  to  manage  these  group  dynamics. 

It  proved  to  be  important  to  state  the  roles  of  the  facilitators  and  the  workshop  participants  clearly.  In 
principle,  the  facilitators  guide  the  process  and  the  participants  take  the  decisions.  Only  on  request  did  the 
facilitators  bring  in  their  own  knowledge  and  experience  in  the  field  of  instructional  design  and  development. 
Sometimes,  however,  giving  examples  proved  to  be  a  good  way  to  get  a  discussion  going  or  to  stimulate  the 
participants  to  consider  alternative  solutions. 

In  three  cases,  the  participants  had  very  high  expectations  of  the  knowledge  available  in  the  area  of 
instructional  design,  illustrated  by  remarks  such  as:  "So,  just  tell  us  which  learning  goals  need  CBT",  "I'm 
sure  there  has  been  done  a  lot  of  research  about  how  to  teach  these  tasks",  and  "There  must  be  ready-made 
solutions  described  in  literature".  The  same  phenomenon  was  observed  in  other  settings,  for  example  during 
evaluations  of  the  MASTER  method  with  target  users  (see  e.g.  van  Rooij  et  al.,  1998;  Verstegen  and  van  der 
Hulst,  2000).  However,  like  other  instructional  design  tasks,  the  development  of  a  needs  statement  is  an  ill- 
structured  problem  for  which  there  is,  usually,  no  guaranteed  best  solution  or  solution  procedure.  Which 
combination  of  training  means  or  instructional  products  is  suitable  depends  not  only  on  educational  factors, 
but  also  on  pragmatic  factors  such  as  the  available  budget,  the  amount  and  capabilities  of  the  instructors,  etc. 
There  are  no  ready-made  solutions  available  that  take  all  these  factors  into  account.  This  was  an  eye-opener 
for  many  participants. 

3.4.  Iteration  and  dealing  with  uncertainty 

In  all  cases  there  was  iteration  within  the  period  dedicated  to  the  development  of  the  needs  statement,  usually 
caused  by  the  fact  that  the  required  input  for  a  later  step  was  incomplete  or  not  elaborated  into  enough  detail 
or  because  of  new  information  from  sources  or  people  not  present  at  the  workshop.  Sometimes,  participants 
phoned  colleagues  during  the  workshop  for  missing  input  information.  In  one  case,  the  participants  proposed 
to  split  of  the  first  phase  (Problem  Statement)  into  a  one-day  workshop,  followed  by  an  interval  to  collect 
additional  information.  The  participants  were  less  inclined  to  iterate  in  order  to  improve  their  design  or  to 
consider  alternative  solutions. 

Uncertainty  played  a  role  in  all  cases,  but  for  different  reasons.  In  the  case  where  the  operational  system  did 
not  yet  exist  there  was  much  uncertainty  about  the  tasks,  especially  the  division  of  tasks  between  team 
members  and  the  learning  difficulty  of  the  tasks.  In  another  case  the  source  of  uncertainty  was  an  ongoing 
reorganisation  with  -as  yet-  unknown  consequences  for  the  division  of  tasks  over  personnel.  In  a  third  case, 
the  participants'  task  was  to  develop  a  needs  statement  for  CBT  to  be  used  in  addition  to  a  simulator  for 
which  the  specifications  were  being  developed  simultaneously  by  other  people.  However,  it  was  not  yet  clear 
what  the  capabilities  of  the  simulator  would  be,  and  indeed  not  even  whether  the  simulator  would  be  bought 
or  not.  In  the  fourth  case,  the  main  goal  of  the  workshops  was  to  develop  guidelines  for  the  application  of 
CBT  in  the  school.  During  the  workshop  an  example  course  was  used,  and  it  was  not  clear  whether  CBT 
would  be  necessary  or  (financially)  feasible  for  this  course.  And  in  the  fifth  case,  most  uncertainty  was 
caused  by  the  fact  that  not  all  stakeholders  were  present  during  the  workshop. 
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4.  Conclusions 

With  the  SLIM  method  used  in  a  workshop  setting  a  needs  statement  can  be  developed  quickly  and 
systematically.  With  experienced  facilitators  the  SLIM  method  is  robust  and  applicable  to  different  kinds  of 
cases  in  different  settings.  During  the  workshops  the  different  aspects  of  the  design  task  were  split  up:  the 
workshop  leader  organised  and  managed  the  development  process,  and  guided  the  development  activities  and 
the  communication  between  the  participants.  A  second  facilitator  was  responsible  for  the  documentation 
process  and  products.  Thus,  the  participants  could  focus  on  bringing  in  their  own  information  and  expertise, 
and  on  making  decisions.  This  proved  to  be  effective  form  of  process  management,  but  requires  skill  and 
expertise  of  the  two  facilitators.  Inviting  all  stakeholders  ensures  that  all  arguments  will  be  taken  into 
account,  and  that  the  selected  solution  will  be  accepted  within  the  organisation.  In  fact,  this  format  can  be 
seen  as  a  combination  of  a  systematic  and  a  relational  or  communicative  approach,  as  advised  by  Kessels 
(1999,  1993),  amongst  others. 

Working  with  a  group,  rather  than  individually,  seemed  to  make  the  process  more  complicated:  there  were 
almost  continuous  'interventions'  caused  by  discussions  between  workshop  participants  or  by  new 
information  brought  in  by  one  of  them.  In  most  cases  the  most  important  stakeholders  were  present,  though 
iteration  was  also  caused  by  information  or  opinions  from  other  people  collected  between  sessions.  The 
SLIM  method  provides  a  framework,  that  ensures  that  all  important  issues  are  systematically  addressed.  It 
also  helps  to  manage  the  iterative  design  process.  When  participants  bring  in  new  information,  the  workshop 
leader  decides  to  discuss  it  immediately  or  to  come  back  to  it  at  a  later  moment.  The  second  facilitator  makes 
sure  that  it  is  documented  with  the  right  step  in  the  report.  Working  with  a  group  also  introduces  a  social 
aspect  to  the  development  process:  during  discussions  not  only  rational  arguments  played  a  role,  but  also 
personal  interests,  personalities,  differences  in  rank  or  power,  existing  conflicts  or  power  struggles  (between 
individual  people  or  between  different  parts  of  the  organisation),  differences  in  interaction  styles,  and  so 
forth.  The  workshop  leader  needs  additional  skills  to  manage  these  group  dynamics.  It  is  possible  that 
GroupWare  systems  could  support  this  aspect  of  the  development  process  during  the  workshops. 
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